LLM-Based Human-Robot Collaboration Framework for Manipulation Tasks

This paper presents a novel approach to enhance autonomous robotic manipulation using the Large Language Model (LLM) for logical inference, converting high-level language commands into sequences of executable motion functions. The proposed system combines the advantage of LLM with YOLO-based environmental perception to enable robots to autonomously make reasonable decisions and task planning based on the given commands. Additionally, to address the potential inaccuracies or illogical actions arising from LLM, a combination of teleoperation and Dynamic Movement Primitives (DMP) is employed for action correction. This integration aims to improve the practicality and generalizability of the LLM-based human-robot collaboration system.Comment: IEEE MHS 202

Visual Tactile Sensor Based Force Estimation for Position-Force Teleoperation

Vision-based tactile sensors have gained extensive attention in the robotics community. The sensors are highly expected to be capable of extracting contact information i.e. haptic information during in-hand manipulation. This nature of tactile sensors makes them a perfect match for haptic feedback applications. In this paper, we propose a contact force estimation method using the vision-based tactile sensor DIGIT, and apply it to a position-force teleoperation architecture for force feedback. The force estimation is done by building a depth map for DIGIT gel surface deformation measurement and applying a regression algorithm on estimated depth data and ground truth force data to get the depth-force relationship. The experiment is performed by constructing a grasping force feedback system with a haptic device as a leader robot and a parallel robot gripper as a follower robot, where the DIGIT sensor is attached to the tip of the robot gripper to estimate the contact force. The preliminary results show the capability of using the low-cost vision-based sensor for force feedback applications.Comment: IEEE CBS 202

Intelligent Detection of Parcels Based on Improved Faster R-CNN

Parcel detection is crucial to achieving automatic sorting in intelligent logistics systems. Most parcels in logistics centers are currently detected manually, imposing low efficiency and high error rate, severely limiting logistics transportation efficiency. Therefore, there is an urgent need for automated parcel detection. However, parcels in logistics centers have challenges such as dense stacking, occlusion and background interference, making it difficult for existing methods to detect parcels accurately. To address the above problem, we developed an improved Faster R-CNN-based parcel detection model spurred by current deep-learning-based object detection trends. The proposed method first solves the false detection problem due to parcel mutual occlusion by augmenting Faster R-CNN with an edge detection branch and adding object edge loss to the loss function. Furthermore, the self-attention ROI Align module is proposed to address the problem of feature misalignment caused by the quantization rounding operation in the ROI Pooling module. The module uses an attention mechanism to filter and enhance the features and uses bilinear interpolation to calculate the feature pixel values, improving detection accuracy. The implementation of the proposed method was validated using parcel images collected in the field and the public dataset SKU110K and compared with four existing parcel detection methods. The results show that our method’s Recall, Precision, [email protected] and Fps are 96.89%, 98.76%, 98.42% and 22.83%, respectively, which significantly improves the parcel detection accuracy

Automatic Camera Calibration Using Active Displays of a Virtual Pattern

Camera calibration plays a critical role in 3D computer vision tasks. The most commonly used calibration method utilizes a planar checkerboard and can be done nearly fully automatically. However, it requires the user to move either the camera or the checkerboard during the capture step. This manual operation is time consuming and makes the calibration results unstable. In order to solve the above problems caused by manual operation, this paper presents a full-automatic camera calibration method using a virtual pattern instead of a physical one. The virtual pattern is actively transformed and displayed on a screen so that the control points of the pattern can be uniformly observed in the camera view. The proposed method estimates the camera parameters from point correspondences between 2D image points and the virtual pattern. The camera and the screen are fixed during the whole process; therefore, the proposed method does not require any manual operations. Performance of the proposed method is evaluated through experiments on both synthetic and real data. Experimental results show that the proposed method can achieve stable results and its accuracy is comparable to the standard method by Zhang

Auditory Feedback for Enhanced Sense of Agency in Shared Control

There is a growing need for robots that can be remotely controlled to perform tasks of one’s own choice. However, the SoA (Sense of Agency: the sense of recognizing that the motion of an observed object is caused by oneself) is reduced because the subject of the robot motion is identified as external due to shared control. To address this issue, we aimed to suppress the decline in SoA by presenting auditory feedback that aims to blur the distinction between self and others. We performed the tracking task in a virtual environment under four different auditory feedback conditions, with varying levels of automation to manipulate the virtual robot gripper. Experimental results showed that the proposed auditory feedback suppressed the decrease in the SoA at a medium level of automation. It is suggested that our proposed auditory feedback could blur the distinction between self and others, and that the operator attributes the subject of the motion of the manipulated object to himself